The test of software products is a very critical activity in any phase of their lifecycle. Generally speaking, every execution of a test of a specific software product is aimed at verifying whether the software product behaves correctly as expected (for example, from the point of view of its functions, requirements and performance). For this purpose, a bucket of test cases is commonly executed on the software product, each one for verifying a specific aspect of the software product. The number of test cases may be very high (up to the order of several thousands), especially in the test of large software applications (such as with a distributed architecture).
In order to facilitate this activity, a test automation infrastructure may be exploited. The test automation infrastructure automatically controls the test by planning the execution of its test cases, capturing corresponding results, and outputting a test report showing them. In this way, the test may be executed repeatedly in a relatively fast way, generally in completely unattended mode. Examples of techniques relating to test automation infrastructures are disclosed in U.S. Pat. No. 6,002,869 and US-A-2005/0204201 (the entire discloses of which are herein incorporated by reference).
The test report provides very useful information to one or more human test analysts in charge of analyzing the test of the software product; for example, the test report typically indicates a progress of the test, the test cases whose execution has failed, when they have been executed, and the like.
However, the analysis of the test report (for example, to infer the causes of the failed test cases and possibly correct them) is a substantial manual task. Indeed, the analysis is generally based on an investigation activity, which requires a heavy human intervention; therefore, this activity is very laborious and time consuming.
The problem is particularly acute when the number of failed test cases is very high. This makes it impractical to analyze all the failed test cases; however, the selection of the failed test cases on which the analysis should focus is completely based on a skill of the test analysts. In any case, the test report may become so complex to be very difficult to interpret.
In order to tackle this problem, some techniques have been proposed for automatically reducing the complexity of the analysis of the failed test cases.
For example, US-A-2007/0006037 (the entire disclose of which is herein incorporated by reference) filters out the failed test cases representative of defects already reflected in historical information; this is based on an automated comparison among defect symptoms associated with the failed test cases and defect symptoms that are known to be not of interest (for example, because due to environmental issues). However, this technique is not very accurate (since it is mainly based on statistical information); moreover, it requires a pre-analysis activity to determine the failed test cases due to environmental issues.
Likewise, US-A-2008/0222501 classifies the failed test cases according to their correspondence with historical failed test cases; for this purpose, attributes of the failed test cases are compared with corresponding attributes of the historical failed test cases, so as to determine recurring failed test cases relating to the same defects according to their degree of matching. However, this technique requires a manual definition of the attributes to be compared; moreover, it does not work efficiently when the software product is very dynamic (since it requires a high manual effort to keep the attributes up-to-date).
At the end, US-A-2009/0265694 selects the failed test cases to be first analyzed according to a corresponding hierarchy; particularly, this selection takes into account whether exercised classes have been recently changed, the relative complexity of the failed test cases and the likelihood that the correction of the corresponding defects may automatically correct the defects of other failed test cases as well.
However, none of the techniques known in the art takes into account the dynamic development of the software product and/or of the test cases.
Moreover, none of them provides any hint towards the possible causes of the failed test cases.